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We apply a distance-based Bell-test analysis method [E. Knill et ah, Phys. Rev. A. 91, 032105 
(2015)] to three experimental data sets where conventional analyses failed or required additional 
assumptions. The first is produced from a new classical source exploiting a “coincidence-time 
loophole” for which standard analysis falsely shows a Bell violation. The second is from a source 
previously shown to violate a Bell inequality; the distance-based analysis agrees with the previous 
results but with fewer assumptions. The third data set does not show a violation with standard 
analysis despite the high source quality, but is shown to have a strong violation with the distance- 
based analysis method. 
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Local realism, the notion that any two non-causal 
events should have no influence on each other (locality), 
and that outcomes should be determined by hidden vari¬ 
ables (realism), is fundamental to classical physics, and is 
a natural view of reality. When Einstein, Podolsky, and 
Rosen noted that quantum mechanics appears to aban¬ 
don local realism, they thought that it must be quantum 
mechanics that is incomplete [T]. Almost thirty years 
later, John Bell showed that local realism and quantum 
mechanics are not only conceptually incompatible, but 
can actually give different statistical outcomes for ex¬ 
periments on entangled particles [2]. The statistical dif¬ 
ferences are quantified via a Bell inequality, a violation 
of which would definitively rule out any local realistic 
theory, thereby ending a central debate of 20th-century 
physics. While entanglement has been experimentally 
demonstrated in various physical systems, due to exper¬ 
imental challenges every Bell test to date has required 
assumptions about either the source and detector (e.g., 
that the detected particles are a fair sample of the to¬ 
tal ensemble emitted from the source), or the possibility 
of signaling between specific events (e.g., assuming that 
there is no signaling between the measuring devices) [3]. 
While these assumptions allow one to make arguments 
against local realism, they present loopholes that could 
be exploited by a local realistic model to violate a Bell 
inequality. 

Furthermore, there can also be implicit assumptions 
within the data analysis itself if it directly or tacitly 
assumes no-signaling or fair-sampling. Even worse, the 
data analysis may directly violate an assumption, thereby 
invalidating the analysis technique. The issue can be 
subtle; for example, in the case of the “coincidence-time 
loophole” [4] , the implicit assumptions can come from an 
otherwise standard coincidence counting method, where 
the coincidence windows are centered on one party’s de¬ 


tection events (the implicit assumption is that the local 
hidden-variable model has no time-dependence) instead 
of using a predefined coincidence window. Finally, addi¬ 
tional loopholes can arise from the assumed source statis¬ 
tics. Two analysis assumptions are noteworthy. The first 
is that most analyses assume that the source emits par¬ 
ticles with independent and identical states. The second 
assumes that the average violation has a Gaussian distri¬ 
bution; in particular, nearly all reported Bell violations 
are cited in terms of numbers of standard deviations of 
violation, whose interpretation requires that the relevant 
distributions are Gaussian for many standard deviations 
from the mean, which fails to hold no matter how many 
particles are detected (for a discussion, see [5]). 

As Bell tests can be a resource for cryptographic pro¬ 
tocols, such as device-independent random number gen¬ 
eration [S] and device-independent quantum key distri¬ 
bution [7], these issues are critical to the security of the 
device, as each loophole allows for an avenue of attack. 
If the device satisfies a loophole-free Bell test, i.e., vio¬ 
lates a Bell inequality with no extra assumptions, then 
the device can be trusted regardless of the manufacturer 
of the device or possible hacking technique. Thus it is 
important to minimize any extra assumptions required 
by the analysis or its interpretation. 

In this paper, we begin by describing how common ex¬ 
perimental Bell tests are performed, and the issues that 
can arise from the data analysis. We then briefly summa¬ 
rize a new, distance-based analysis technique described 
in Ref. [5], and in the subsequent sections we compare 
this technique to the conventional analysis for real data 
sets from three distinct experimental configurations; one 
is the first demonstration of a system capitalizing on 
the coincidence-time loophole to fake a Bell-inequality 
violation, while the other two are a pulsed version and 
a continuous version of the quantum source presented 
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in Ref. [5]. Finally, we discuss general features of the 
distance-based analysis technique that apply to all Bell 
test experiments. 


I. EXPERIMENTAL BELL TESTS 

An idealized (bipartite) Bell test consists of a series of 
“trials”. For each trial a pair of quantum systems is pre¬ 
pared and distributed to two parties, Alice and Bob, who 
independently and randomly choose a setting at which 
to measure their quantum system. For the experiments 
discussed here, the quantum systems are photons, and 
the settings are chosen from two possibilities, labeled 0 
or 1. Alice and Bob’s measurement settings choices are 
denoted by and s'®, respectively; their corresponding 
measurement outcomes are denoted by and re¬ 
spectively. Since sufficiently high efficiency detectors are 
still difficult to obtain, recent experiments have employed 
only one detector on each side, so the possible outputs 
are either a detection event (which we denote by t^A = 1 
for Alice, similarly for Bob) or the absence of a detection 
event (denoted by t\ = 0). After many trials, Alice and 
Bob compare their results to determine how the detec¬ 
tions are correlated with their joint settings choices, for 
example, how often they saw a coincident detection given 
a pair of settings. 

One type of Bell test experiment in high-efficiency sys¬ 
tems uses polarization entangled photons liiin]. To gen¬ 
erate the entangled photons, a strong pump laser passes 
through a nonlinear crystal setup, where each photon of 
the pump has a small probability of downconverting into 
a pair of entangled photons, one of which is sent to Alice, 
and the other to Bob. The measurement settings for the 
Bell test are provided by a polarizer placed after either 
a half-wave plate in a rotation mount, or a Pockels cell. 
Afterwards, the photons are detected on separate high ef¬ 
ficiency photon counters, e.g., transition-edge-sensor de¬ 
tectors (TES) [TT]. The detection events from each TES 
are recorded by a time-to-digital converter, and the re¬ 
sulting timetag sequences are saved for later analysis to 
check for correlations. A new timetag sequence is saved 
for each new setting that Alice and Bob choose. Be¬ 
cause motorized rotation mounts (and Pockels cells to a 
lesser extent) cannot always change settings quickly, it is 
possible that multiple detection events occur before the 
settings can be changed. Eor example, the two recent 
photon experiments kept the same setting for 1 s [5] and 
300 s [TU] intervals. Therefore, multiple conventional one- 
photon-pair trials are performed at the same setting, so 
these trials cannot be considered strictly independent of 
each other. This dependence issue can be fixed by dis¬ 
carding all but the very first trial for a given setting (at 
the cost of much less usable data), or if the data anal¬ 
ysis considers all events taking place while the settings 
are held constant to constitute one trial instead of many. 
The latter approach is discussed in the following section. 

In the case where a trial is intended to consist of the 


measurement of only one photon pair, determining pre¬ 
cisely when a given trial occurs can be difficult. Eor ex¬ 
ample, all single-photon detectors have an intrinsic un¬ 
certainty of the arrival time of the photon. Further¬ 
more, downconversion is a probabilistic process, where 
the emission can occur at any time when the pump 
laser has a non-zero amplitude. This is most notable 
for continuous-wave lasers, where downconversion events 
happen randomly, uniformly in time. For each trial Alice 
and Bob need to determine if they have a coincidence 
event (both saw a detection event), a single event (only 
one saw a detection), or neither saw a detection event; 
they must determine which type of event occurred de¬ 
spite the temporal uncertainty of their individual events. 
(The standard Bell test analysis can be modihed so that 
it is not necessary to account for the cases where neither 
party detected a photon.) For example, typical quantum 
optics experiments determine coincidence events by al¬ 
lowing for a coincidence window around one party’s - say 
Alice’s - detection events: if Bob has a detection event 
within the coincidence window determined by her detec¬ 
tion event, then it is called a coincident detection. 

In a Bell test, however, this seemingly reasonable 
method for determining coincidences cannot exclude all 
local realistic models, as it opens up a loophole that can 
be exploited by a hacker to produce an apparent Bell 
inequality violation without any quantum correlations. 
The loophole, called the coincidence-time loophole, al¬ 
lows for a time-dependent local hidden-variable model [4] . 
Consider the Clauser-Horne (CH) Bell parameter [H] in 
the form 

BcH = PAsito = l,t® = I) +PABitQ = l,tf = 1) 

+ PAB{tf- = = 1) -pAB{ti = I,tf = I) 

-PA{to =l)-pBito =1), ( 1 ) 

where tf = 1 (t® = I) is a detection event for Al¬ 
ice (Bob) with z being the measurement setting for Al¬ 
ice’s (Bob’s) detector, pAB{x,y) denotes the settings- 
conditional probability of the joint outcome of x and y 
for Alice and Bob’s detectors, respectively, and pa{x) 
(pb(x)) is the setting-conditional probability of outcome 
X for any given trial at Alice’s (Bob’s) detector. Then it 
can be shown that — 1 < Bch < 0 for any local realistic 
model. 

Consider an experiment where the times of photon- 
pair arrivals at the two parties are unknown. Normally, 
we assume that the pairs arrive at a constant rate rp. Let 
ta^x), rp^x) and rAsix^y) be the rates of events whose 
detection probabilities are determined by pa(x), pb{x) 
and pAB{x,y). Given the constant rate assumption, we 
can express pAB{x,y) = rAB[x,y)/rp and similarly for 
the other rates. The quantity Bch can be inferred ac¬ 
cordingly and whether or not it is violates the inequality 
Bch < 0 does not depend on the rate rp. Thus it is not 
necessary to know the rate to observe such a violation. 

To exploit the coincidence loophole, a hacker who has 
full control of the photon source can, at random times at 
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a rate of rn , send a group of four pulses (two to Alice and 
two to Bob as shown in Fig. with each pulse offset by 
a little less than the Alice-detection-centered coincidence 
window used by Alice and Bob. In doing so, the pulses 
that result in detections for settings = 1 and = 1 
are separated by nearly three coincidence window “radii” 
and therefore do not result in any coincidence counts, 
whereas at every other setting combination the detected 
pulses fall within the coincidence window. Consequently, 
a hacker can achieve an apparent Bell violation of up to 
Bch = fu/fp > 0, given the experimenter’s assumed 
photon-pair rate rp. If the experimenter attempts to 
measure rp independently, this measurement may also 
be subject to the hacker’s manipulations. The trial data 
set itself only yields lower bounds on rp. That is, assum¬ 
ing (wrongly) that the detections arise from constant- 
rate photon pairs, we have that rp should be at least 
the sum of the rate of detections by A and B, minus the 
rate of coincident detections at any given setting com¬ 
bination. This rate is maximized for settings = 1 
and S'® = 1, where it is 2r/f. Accordingly, rp > 2rH- 
Setting rp = 2rp gives a maximum inferred violation of 
Bch = 1/2 per presumed pair, which exceeds the maxi¬ 
mal quantum mechanically allowed value of Bch ~ 0.207 
and matches the maximum allowed by no-signaling |13j . 
When quantifying the violation in Fig. and Fig. and 
in the discussion in App. we use this normalization 
for the values of Bch, that is we set rp = 2r/f. 

One method to prevent the coincidence loophole is to 
provide synchronization pulses to both parties, which 
define the trials independently of any detection events, 
as was done in Ref. [9]. Another option is to perform 
a distance-based analysis of the data, which we now 
present. 


II. ANALYSIS DESCRIPTION 

A high-level explanation of the coincidence-time loop¬ 
hole is that the non-local method for inferring trials in¬ 
validates the assumptions underlying the Bell inequali¬ 
ties. The solution is to ensure that each party knows 
in advance the time and duration of a trial and relates 
recorded data accordingly. Moreover, if the settings are 
held hxed over multiple trials, it is necessary to make ad¬ 
ditional assumptions; for example, one can assume that 
the trials are independent and identical. To avoid making 
such additional assumptions, one should predefine trials 
such that they are associated with the time intervals be¬ 
tween making random settings choices. (An alternative 
using party-dependent coincidence window sizes is de¬ 
scribed in Ref. [S].) For the experiments analyzed here, 
this means that each party’s measurement outcome is 
their entire timetag sequence recorded between making 
settings choices, rather than a single detection or non¬ 
detection. Thus, the complete results from a trial consist 
of each party’s settings choice and the timetag sequences 
they measured before the next setting was applied. Note 
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FIG. 1: A diagram illustrating the coincidence-time loop¬ 
hole. Here = 1 (t^ = 0) corresponds to a detection event 
(no detection event) when Alice chooses measurement setting 
= z, and similarly for Bob. In diagram i, a coincidence 
window (black arrow) is selected based on Alice’s detection 
event. A hacker can exploit this loophole by staggering pulses 
in time as shown. In this case, if the radius (half width) of 
the coincidence window is between T and ST, there are no 
{s^ = l,s® = 1} coincidence counts, implying that Bch in 
Eq.[^is greater than 0, even for a classical source. In diagram 
u, a well-defined trial is used, where the window is centered on 
a synchronization pulse (blue line); the loophole vanishes as 
there is no longer a way to address only the = 1, s® = 1} 
coincidence term. If a large coincidence window (solid ar¬ 
row) is used, then every measurement setting has a coinci¬ 
dent event, resulting in Bqh = 0. If a short trial window 
(dashed arrow) is used, then there are only coincident and 
single events at settings = 0, s® = 0}, giving a Bell value 
of Bch = —1. 

that in the absence of large separations between A and B, 
this may make it difficult to ensure locality by space-like 
separation of relevant events. In two of the experiments 
below we can exclude all local realistic probabilistic mod¬ 
els, but in principle, a hacker could have exploited the 
ability to communicate settings between A and B before 
the end of a trial to effect arbitrary, non-local-realistic 
probability distributions. 

Generalizing the notation introduced above, we de¬ 
note the timetag-sequence measurement outcomes of the 
two parties by and tfe, where denotes A’s out¬ 
come with the subscript indicating the setting used, 
and similarly for B. Since the settings choices are un¬ 
der experimenter control, their probability distribution 
is known. For the Bell tests considered here, each of the 
four setting-choice combinations has probability 1/4. 

To review the principles of the analysis method in [5] , 
consider first a general local-realism test. The method 
begins by constructing a Bell function B of trial results 
such that a Bell inequality in the form 

{B{tfA,tfn,s^,s^))i^n<0 (2) 

holds for all local realistic models. Here, (.. .)lr denotes 
the expectation with respect to a local realistic probabil¬ 
ity distribution, where the settings distribution is fixed 
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as above. Given such a Bell function, a violation can be 
demonstrated in an experiment by showing a statistically 
significant positive value for an empirical estimate B of 
B = {B{tfA,tfB,SA,SB))EX, where (.. .)ex denotes the 
expectation with respect to the experimental probability 
distribution. The traditional method for evaluating sig- 
nihcance is via the sample standard error of B. This can 
be used to assign approximate confidence intervals for 
B but cannot quantify the extremely high significance 
of the evidence against local realism that we seek. To 
quantify the significance, it is desirable to determine p- 
value bounds in the framework of statistical hypothesis 
testing. Ref. [S] shows how to systematically use lower- 
bounded Bell functions to obtain such bounds from the 
trial results. 

A general strategy for constructing Bell functions that 
can be used for conservative estimates of B and p-value 
bounds is given in Ref. [8j. Here, “conservative” means 
that the estimates and bounds are statistically valid with 
no approximations or extra assumptions on distributions 
other than the standard ones, namely that the settings 
probabilities are known and that local realistic distribu¬ 
tions are mixtures of outcomes determined by the lo¬ 
cal settings. The fundamental principle is to start with 
settings-dependent “distance” functions on 

the measurement outcome pairs; such functions are re¬ 
quired to satisfy a generalized, twice-iterated triangle in¬ 
equality 


(If I is non-negative and independent of the settings, then 
this is the conventional twice-iterated triangle inequal¬ 
ity. Here we use the term “distance function” to refer to 
any function family I satisfying Eq. H) Since local re¬ 
alistic models are given by probability distributions over 
deterministic models where a party’s setting determines 
the party’s measurement outcome, a Bell function can be 
constructed from I according to 


B{tfA,tfB,SA,SB) = 


—h,oiti, to) 
—to,o{to , tQ ) 
—lo,i{to ) tf ) 


if SA = 1, SB = 1, 
if sa = 1, s_b = 0, 
if SA = 0, SB = 0, 
if SA = 0, SB = 1. 

(4) 

The use of distance functions to obtain Bell inequalities 
was introduced by Schumacher in Ref. m- 

For timetag sequence outcomes associated with exper¬ 
iments that are intended to violate a CH-type inequality, 
Ref. [5] shows that one can dehne distance functions ac¬ 
cording to a minimum cost of converting the first timetag 
sequence into the second by shifting and/or deleting 
timetags. A feature of the technique is that in the limit 
where the average time between detections is large com¬ 
pared to the time-jitter (the uncertainty in the time of the 
detection), the distance function can be made to match 
the value of any CH-type Bell function. One issue is that 
the costs defining the distance function are parametrized, 
and we wish to choose these parameters optimally given 


the characteristics of the experiment. However, to avoid 
biases and remain conservative, it is necessary to choose 
the parameters beforehand, independent of the data to 
be analyzed. That is, contrary to what is often done in 
experiments, no part of the “final data” can be used to 
find analysis parameters, such as delays. Otherwise the 
validity of confidence intervals or p-values is lost. The 
parameters can instead be determined by setting aside 
a fraction of the trials from the beginning of the exper¬ 
iment. This “training data set” is used for optimizing 
analysis parameters. The remainder of the trials consti¬ 
tute the analysis data set and should only be analyzed 
once the parameters have been chosen. In the applica¬ 
tions below, the training set serves to determine two Bell 
functions. The first is designed to maximize a CH-like 
violation and can be compared to traditional (that is, 
non-distance-based) measures of violation. For report¬ 
ing these violations, we modify the conventional method 
so that the violation reported is meaningful without as¬ 
suming that the trials are independent, as explained in 
App. The second Bell function is a systematically 
“truncated” version of the first; the truncation method 
is general and can be applied to any distance-based Bell 
function [5]. The second Bell function is bounded, so we 
can apply the techniques from Ref. [5] to obtain a p-value 
(upper) bound. As these p-value bounds are extremely 
small, we give their negative logarithm base 2, called the 
log 2 -p-value (lower) bound. See Sect. IV for the inter¬ 
pretation of p-values and their comparison to Gaussian 
tails. 

Because all the experiments discussed below were per¬ 
formed before the statistical techniques were fully devel¬ 
oped, their analysis was retrospective and in this sense 
deviated from the ideal protocol; the deviations are dis¬ 
cussed in App. 


III. EXPERIMENTAL REALIZATION OF THE 
COINCIDENCE-TIME LOOPHOLE 


We realized the coincidence-time loophole experimen¬ 
tally by combining two attenuated lasers on a beam split¬ 
ter for both Alice and Bob (Fig. [^. For Alice, one laser 
is polarized orthogonal to the polarizer setting for mea¬ 
surement setting s^ = 0, while the other laser is polar¬ 
ized orthogonal to the polarizer setting for = 1, and 
similarly for Bob. This allows the source to address the 
measurement settings independently (i.e., when we send 
a laser pulse polarized along (s"^ = 1)-*-, we should only 
receive detection events for measurement setting s^ = 0). 
We then attenuate the sources to a mean photon num¬ 
ber per pulse of around 10. The relatively high mean 
photon number offsets the loss in the measurement and 
detection process, but is still small enough to minimize 
the effect of crosstalk in the polarizer (there is a small 
chance that the polarization state to be blocked is still 
transmitted through the polarizer). We then pulse the 
lasers as shown in Fig. with adjacent pulses separated 
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by T = 1 IIS. If we determine the number of coincidence 
events by checking if Bob had a detection event within a 
window (e.g., 2/is) around Alice’s detection events, then 
we observe Bell inequality violations up to Bch = 0.49 
(with the normalization discussed earlier), where Alice 
and Bob use the optimal settings for an ideal maximally 
entangled state as specihed in the caption of Fig. A 
plot of the data analyzed in this way is displayed in Fig.[^ 
We see a “violation” of over 2700 -ct (assuming Gaussian 
statistics). By altering the two laser polarizations and 
increasing the mean photon number to offset any addi¬ 
tional losses, we have been able to exploit this loophole 


for a wide range of measurement settings, see App. A1 


In addition, the degree of violation can be altered by 
changing the laser polarization. As a final note, while 
the plot in Fig. has a well-defined structure, it is possi¬ 
ble to broaden the observed “violation range” by proba¬ 
bilistically switching between local hidden-variable mod¬ 
els with different pulse spacings; therefore, one cannot 
simply look at a plot of the Bell violation versus coinci¬ 
dence window size to determine if the coincidence-time 
loophole is being exploited. 

In contrast, if we use a predefined coincidence window 
centered on a predefined time rather than one centered 
on a detection (see App. A 2 for details), we do not see a 
statistically significant Bell violation, as shown in Fig. 
Furthermore, when we use the distance-based analysis 
from Ref. [5], the results correctly do not indicate that 
the system is behaving contrary to local realism. Ini¬ 
tially, in the training set, where delays are determined 
to offset electronic latencies, the delays on apparent co¬ 
incidences were found to depend highly on the measure¬ 
ment settings, due to the scheme for exploiting the coin¬ 
cidence loophole. From the other experiments using the 
same setup, the latencies are known to be small, so for 
demonstration purposes we ignored this observation and 
did not offset for electronic latencies. The distance-based 
Bell function (Eq.|^ is then significantly negative, show¬ 
ing no evidence against local realism according to this 
analysis. 

While the data set in this case is contrived to be clearly 
determined by a local hidden-variable model, in real ex¬ 
periments the issues are far more subtle. For example, 
avalanche photo-diodes can have a count-rate-dependent 
latency, and since each measurement setting can have dif¬ 
ferent detection rates (for example, in Ref. [3] , the count 
rates differed by a factor of 3), it is critical that the anal¬ 
ysis is not susceptible to these minor latency shifts. To 
show that these issues are relevant, Ref. [5] presents a 
coincidence-loophole-exploiting scheme whose statistics 
closely match those of a standard photon-pair source. 


IV. EXPERIMENTS WITH VIOLATION 

The example above shows the use of the distance-based 
analysis technique to “catch” an invalid violation of a Bell 
inequality with a purely classical source. The following 



FIG. 2: A diagram of our experimental setup to produce 
the local hidden-variable model described in Fig. On Al¬ 
ice’s side, we electrically pulse two lasers with a pulse width 
of 100 ns; these pulses then pass through polarizers aligned 
orthogonally to her two measurement settings (Pol (s'^ = 0)^ 
and Pol (s^ = 1)^). That is, we emit pulses that will not 
pass through one of the two measurement settings, ensuring 
only one of the measurement settings will detect our optical 
pulse. The laser pulse that passes through the (s'’^ = 0)^ 
polarizer is emitted 2/rs before the (s'^ = 1)^ laser pulse. We 
attenuate the lasers enough so that after they are combined 
on a beam splitter, each pulse has a mean photon number 
of approximately 10, to offset any system loss while mini¬ 
mizing the noise due to crosstalk in the polarizers. Similarly 
on Bob’s side, we combine two attenuated lasers on a beam 
splitter. Here, the (s® = 1)^ pulse is emitted 2 fis before the 
(s® = 0)^ pulse. Both are offset from Alice’s photon pulses 
by 1 IIS. The basis choice for the polarization analysis is im¬ 
plemented with a half-wave plate (HWP) and polarizer (Pol), 
where the settings are —11.25° for = 0, 33.75° for = 1, 
11.25° for = 0, and —33.75° for s® = 1 (corresponding to 
the optimal CH-Bell-inequality-violating settings of a perfect 
maximally entangled state). The photons are then detected 
by an avalanche photo-diode (APD), with an efficiency lower 
than 66 %, the outputs of which are recorded using a time-to- 
digitaf converter. The results of analyzing the data both with 
a coincidence window determined by Alice’s detection event, 
as well as a predefined coincidence window, is displayed in 
Fig. [3] 


two examples demonstrate the strength of this analysis 
on data with actual quantum correlations. 

First, we consider the data collected and analyzed in 
Ref. [S], where the experiment had a high enough sys¬ 
tem efficiency and low enough noise to be able to violate 
a CH Bell inequality. The data set was taken with an 
external clock synchronized to a laser pulse. A prede¬ 
fined coincidence window of 2.4 ^s was centered around 
the laser synchronization pulse, from which a trial could 
be well defined, avoiding the coincidence-time loophole 
discussed above. The data set was collected by changing 
the measurement settings randomly every second, collect¬ 
ing for 4450 different measurement setting choices. For 
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FIG. 3: Two plots of the measured Bell parameter, Bch 
(Eq. 0 (solid blue line), as a function of the coincidence win¬ 
dow radius for our experiment using a classical source to pro¬ 
duce the local hidden-variable model shown in Fig. When 
the data set is analyzed with a coincidence window deter¬ 
mined by a detection event (diagram i), the coincidence-time 
loophole can be exploited to produce a Bell violation (values 
greater than 0). We separated each pulse by 1 /rs, so with this 
model we see Bch > 0 for any coincidence window radius be¬ 
tween 1 fis and 3/is. For coincidence windows less than 1 fis, 
we do not have any coincidence counts, but we still have single 
counts, resulting in a negative Bell parameter of Bch > — 1. 
While this value depends on the chosen normalization, the 
minimal inferred value of Vp is at Vp = 2rh, resulting in the 
most negative Bell parameter of —1. With window sizes larger 
than 3 /rs, we assign coincident and single events to nearly ev¬ 
ery detection event, resulting in a Bell parameter of 0. In 
contrast, analysis with a fixed predefined coincidence window 
(diagram li), or using the technique described in the text, 
never produces a violation for any window size. When the 
data set is analyzed with a predefined coincidence window, 
the Bell parameter remains between -1 and 0, and therefore 
does not show a violation of local realism. The results match 
well the predictions given the structure of the classical source 
as explained in App. |A 2[ The positions of the transitions 
are due to the location of the predefined coincidence window 
relative to the pulse set. The transitions between 0 and — 1 
are not sharp because of the slow desynchronization between 
the fixed windows and the actual source pulse rate. That is, 
the window slowly drifts such that it is not always centered 
on the pulse set. For more details, see App. |A 2[ 


the analysis in Ref. |^, the data set was partitioned into 
50 different Bell tests. The uncertainty was calculated 
from the distribution of the 50 different Bell parame¬ 
ters using the sample standard error. The reported value 
from this approach was Bch = 5.4 x 10“® ± 7.0 x 10“®, 
a 7.7-a violation, where the conventional interpretation 
of the large violation assumes Gaussian statistics. In 
contrast, here we analyze the same data set with the 
distance-based method without making distributional as¬ 
sumptions or approximations. Additionally, whereas the 
previous analysis required each pulsed trial to be inde¬ 
pendent and identical to allow for the settings being fixed, 
the new analysis, detailed in App. treats each 1 s pe¬ 
riod with fixed settings as one trial and therefore does not 
require this assumption. We find a log 2 -p-value bound 
of 33, which means that for every local realistic model, 
the probability that this analysis reports a log 2 -p-value 
above 33 is less than 2“^^ = 1.16 x 10“^°, a very unlikely 
event. While this result is equivalent to a 6.3-cr violation 
for Gaussian statistics (we give the Gaussian-equivalent 
violation only for comparison; it is computed from the 
p-value bound of 2“^^ by solving j = 2“^^ 

for x), slightly lower than the 7.7-a violation reported in 
Ref. [5], it does not assume Gaussian statistics. Thus, 
we see that with minimal degradation of the evidence for 
Bell-inequality violation, we have reduced the required 
assumptions on the system: the trials do not need to 
be independent and identical and the distributions are 
not approximated by Gaussians. If the system is being 
hacked, lack of independence and Gaussianity are even 
more pronounced. 

Finally, we consider a different data set taken on the 
same high-efficiency system, but with neither pulsing the 
laser nor having an external clock. To analyze the data 
conventionally, we partition time into segments indepen¬ 
dent of the data (that is, we impose a fixed coincidence 
window). Since we are not determining the coincidence 
window based on the data, it is not susceptible to the 
coincidence-time loophole (see Fig. [^i). However, be¬ 
cause we are introducing a coincidence window that is 
not related to the arrival time of photons, due to the de¬ 
tector time-jitter, we are effectively introducing loss into 
the system. That is, if the window is small compared to 
the time-jitter, then it is possible that Alice and Bob’s 
detection events from a single pair of photons are nev¬ 
ertheless registered in different time segments, resulting 
in two single counts without a coincidence count. In the 
opposite limit, the window becomes too large, which re¬ 
duces the Bell parameter due to the high likelihood of 
counting uncorrelated photon pairs as coincidences. The 
result of analyzing the data in this way is displayed in 
Fig.i While the source quality is sufficient for a Bell 
test (that is, it has high heralding efficiency and high en¬ 
tanglement quality), the effective loss introduced by this 
conventional analysis is too much for us to adequately ex¬ 
tract the quantum correlations. If we instead analyze the 
data using the distance-based approach discussed here, 
we find a violation with a log 2 -p-value bound of 269, the 
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FIG. 4: A plot of the CH-Bell parameter from a non- 

pulsed experiment, analyzed using different predefined coin¬ 
cidence windows. While the system is capable of a detection 
loophole-free violation as verified in Ref. [5], the inefficien¬ 
cies of a conventional analysis with predefined coincidence 
windows are unable to produce a non-classical result: for 
small window sizes, the analysis introduces loss because of the 
timing uncertainty (time-jitter) of the detectors, causing the 
mis-assignment of some detection events as non-coincidence 
counts; large windows increase the system noise, to which the 
CH Bell inequality is very sensitive, again resulting in a re¬ 
duced Bell parameter. Here we show the Bell parameter for 
many different coincidence windows, the blue solid line is a 
fit to all of the data points (the points are spaced by 10 ns), 
each of which lie within the thickness of the line. With con¬ 
ventional analysis, we do not observe a Bell violation (above 
the red dashed line) for any coincidence window. With the 
new analysis discussed in [8], we observe a violation with a 
log 2 -p-value bound of 269. 

equivalent of a 19-cr violation (see App. for more de¬ 
tails). In addition to revealing a violation where con¬ 
ventional analysis would not produce one, the confidence 
in the violation is actually significantly larger than that 
with the pulsed source presented in Ref. [9]. This is be¬ 
cause we can utilize a system that is “on” more often than 
a pulsed source (which by definition has no data collec¬ 
tion between pulses), thereby resulting in substantially 
more data. 


V. DISCUSSION 

As shown in the above examples, the distance-based 
analysis of Ref. [5] is able to improve the statistical 
significance of a Bell inequality violation, as well as re¬ 
duce the required assumptions compared to a standard 
analysis. While the analysis uses distance functions as a 
measure of the violation, it has important features com¬ 
mon to any conservative analysis of Bell inequality data. 
First, to estimate the significance of the violation, it is 
important to use p-value bounds instead of standard devi¬ 
ations. The latter are unreliable for the high significance 


of typical Bell inequality violations. Second, to prevent 
overestimating the statistical significance of the Bell in¬ 
equality violation, delays, coincidence windows and other 
such analysis parameters should be determined from a 
training data set (that is then discarded) rather than the 
data to be analyzed. Otherwise, if the final data set is 
used to determine these parameters, the reported viola¬ 
tion may be biased by statistical fluctuation rather than 
reflect a fair estimate. Finally, all Bell tests should have 
predefined trials to avoid opening up additional loopholes 
(e.g., the coincidence-time loophole). The predefined tri¬ 
als may be based on a timetag sequence according to the 
chosen settings as presented here, specific laser pulses 
detected on a photodiode as presented in Ref. [9], or the 
detection of heralding photons as in the ion experiments 
of Ref. . 
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Appendix A: Additional Experimental Details 

This section further discusses our classical source that 
exploits the coincidence time loophole. The first subsec¬ 
tion explains how the source can be tuned to match Alice 
and Bob’s expectations (i.e., to give violations consistent 
with quantum mechanics). In the second subsection we 
use a predefined coincidence window to analyze the data 
from the classical source and find no violation of local 
realism. 


1. Controlling Violation Size 

In an actual attempt of a Bell test, Alice and Bob 
would likely suspect the presence of a hacker if their es¬ 
timated CH-Bell parameter is beyond the quantum me¬ 
chanical limit of {\/2 — l)/2. Even more so, if Alice and 
Bob know that they have low system efficiencies, then 
the value they expect is well below {V^ — l)/2. In par¬ 
ticular, with low efficiency, Alice and Bob design their 
system to use states of the form cos 9\HH) -|- sin0|W) 
(see Ref. [Sllinille]) to maximize the measured violation. 
Consequently, a hacker would want Alice and Bob to be¬ 
lieve that they prepared a less entangled state (states 
with 6 farther from 7r/4). If Alice and Bob estimate 9 
for their state, they obtain a maximum Bell parameter 
they expect. Ideally, the hacker controls the measured 
Bell parameter to match Alice and Bob’s expectation and 






FIG. 5: A plot of a few measured data points from our 
classical source when analyzed incorrectly (susceptible to the 
coincidence-time loophole discussed in Sect.[T|. The blue curve 
is the predicted quantum mechanical maximum given the 
state cos9\HH) + sm9\VV). Here, we assumed Alice and 
Bob have a target 9 and use the optimal measurement set¬ 
tings for the input state. We then adjusted the input polarizer 
angle to match the quantum mechanically allowed maximum 
Bell parameter, given Alice and Bob’s measurement settings. 
The results of a few measurements are displayed as the red 
data points, which are indistinguishable from the quantum 
mechanical expectation. 

avoid suspicion. In our case, with the source depicted in 
Fig. § we can tune the input polarizers (and adjust the 
laser diode brightness to compensate the increased loss) 
to create nearly any value of the Bell parameter. The 
results of a few measurements using this technique are 
displayed in Fig.[^ 

2. Predefined Window Anaiysis 

To use a predefined window to analyze the data ex¬ 
ploiting the coincidence-time loophole, we first add in a 
synchronization signal at the rate equal to the rate that 
the source emits a set of pulses, 100 kHz in our case. 
As there was no actual synchronization signal when the 
data set was taken, we implement this signal in post¬ 
processing. For comparison with Fig. where the pre¬ 
defined coincidence window is in the center of the pulse 
set, we placed the first synchronization signal in the cen¬ 
ter as determined by the first two detection events in 
the data set. We then create a periodic signal by spac¬ 
ing each synchronization signal by 10 fxs (= 1/100 kHz). 
To compensate for the relative temporal drift between 
the function generator and the timetagging electronics, 
we reset the synchronization signal every 500 detection 
events to be re-centered in the pulse set. If the separa¬ 
tion between adjacent pulses is 1 ^s (see Fig. [^, then 
we would expect a Bell parameter close to 0 for windows 
less than 0.5 /xs, since there will be neither single nor 
coincident events (other than occasional dark counts, no 


event will fall within the predefined window). For win¬ 
dows between 0.5/xs and 1.5/xs we would expect a Bell 
parameter close to —1, since we see events primarily from 
= 0,5-® = 0}. That is, PAsito = = 1) = 1, 

PA{to = 1) = 1, and PB^tQ = 1) = 1 in Eq. while all 
other terms are 0. Finally, for predefined window sizes 
larger than 1.5 /xs, all terms in Eq.j^are equal to 1, lead¬ 
ing to a Bell parameter of 0. The results of analyzing 
the classical data with a predefined coincidence window 
of variable width are displayed in Fig. 

Appendix B: Discussion of Analyses 

Here we describe in detail the distance-based analyses 
of the data from the three experiments discussed in the 
paper. The results reported are from final analyses that 
adhered to the protocol of inferring parameters from the 
training set and applying them adaptively to the anal¬ 
ysis set. However, the final analyses were not strictly 
blind; the data set was available for some time while our 
analysis methods were being developed and there were 
multiple early analysis attempts involving various tech¬ 
niques. Features of the data discovered in these attempts 
required changes in preprocessing and strategy. These 
changes are described below as needed. 

Each data set was analyzed by two or three meth¬ 
ods for comparison purposes. The simplest method is 
a conventional analysis based on coincidence counting. 
The results of this method are susceptible to the coin¬ 
cidence loophole and require strong assumptions on the 
source and its statistics. The second method involves 
our distance-based Bell-function analysis applied to trials 
consisting of all the data acquired while the settings were 
held fixed. The third computes “certificates” of violation 
(given as log 2 (p)-values) using the prediction-based-ratio 
(PBR) protocol [SI Hlj with truncated versions of the 
distance-based Bell functions. We discuss the analysis of 
the three experiments in reverse order, which is also the 
order in which the data sets were received and analyzed. 


1. Continuously Emitting Quantum Source 

The data set for this experiment consists of 3953 trials 
with randomly chosen settings. The measurement out¬ 
comes consist of a sequence of timetags for each party, 
where each timetag records a detection event. The av¬ 
erage numbers of recorded detections per trial are ap¬ 
proximately 1400 on setting 1 and 4900 on setting 2 for 
both parties. Each trial’s results are stored in one file. 
The files for eight trials were corrupted and therefore 
discarded, leaving 3945 trials. The timetag sequences 
were preprocessed in two steps. The preprocessing pa¬ 
rameters were determined at an early stage of analysis 
with a set consisting of 97 x 4 randomly chosen trials 
with 97 at each settings choice. (The final analysis was 
performed in the order in which the experiment was per- 




9 


formed with the initial trials used for training-see below. 
The preprocessing parameters suggested by the training 
set for the final analysis were the same up to statisti¬ 
cal fluctuations, so we did not change them for the fi¬ 
nal analysis.) The first preprocessing step compensated 
for transient artifacts near the beginning and end of the 
timetag sequences. We therefore used only timetags from 
the middle portion of the sequence determined as follows: 
The sequence durations are approximately 1 s. For each 
trial, we first determined the earliest recorded time to 
in both parties’ sequences and set tg to be the second 
multiple of 10® past to (in the time units used for the 
timetags, 156.25ps) Thus tg = ([tg/10®J -I- 2)10®. We 
then used only timetags with recorded times t satisfying 
to — t < tg -I- 6 X 10®. We remark that this prepro¬ 
cessing step is non-local, which is in general undesirable. 
We are not aware of any way in which a local realistic 
(LR) source could exploit this, though the possibility ex¬ 
ists. The second preprocessing step corrected a system¬ 
atic timing offset between the recorded times for the two 
parties. The offset was applied to all timetag sequences 
of party A and involved shifting the timetags by —685 
time units (i.e., 107ns). For comparison, the time-jitter 
determined as the typical distance between apparent co¬ 
incidences is of the same order. 

All analysis attempts used the preprocessing of the pre¬ 
vious paragraph. We describe the final analysis first and 
then discuss how we arrived at the final analysis. For the 
conventional analysis, we used the hrst 197 trials to de¬ 
termine the optimal coincidence window. We then com¬ 
puted the number of coincidences for each trial. The co¬ 
incidences were determined as described in [5] rather than 
with the simple Alice-centered windows used in the main 
text. We then computed the estimated total violation as 
described in App. [C| according to the Bell-inequality used 
for the original analysis of the pulsed quantum source in 
Ref. [9]. The total violation according to this analysis 
is 5.14(10) X 10^, corresponding to a nominal signal-to- 
noise ratio (SNR) of 59.7. The latter is the ratio of the 
total violation to the estimated uncertainty; see App. 

The distance-based analysis was performed adaptively. 
An adaptive procedure was required because the parame¬ 
ters of the distance-based Bell function used are sensitive 
to the apparent drifts in count-rates over time. Start¬ 
ing at the 201st trial and then every 400 trials, we re¬ 
optimized the Bell function parameters on the previous 
800 trials. (Before the 801st trial we used all the tri¬ 
als already processed, including the Hrst 200.) The Bell 
function was then computed for each of the next 400 
trials, before re-optimizing the parameters. We then es¬ 
timated the total violation as described in App. 0 The 
total violation according to the distance-based analysis is 
2.52(10) X 10"^, corresponding to a nominal SNR of 24.5. 

The Bell-function values from the distance-based anal¬ 
ysis were then used in an adaptive version of the PBR 
analysis. This required adaptively computing the param¬ 
eters for Bell-function truncations and the mixtures used 
in constructing the test factor according to the protocol 


in Ref. [5]. This proceeded similarly to the distance- 
based analysis, except that the parameters were updated 
every 10 trials and optimized on the last 400 (or less) tri¬ 
als. We chose the more frequent update because there is 
little computational cost in doing so and the truncation 
and mixture parameters are sensitive to small drifts in 
the conditional means of the Bell function. The log 2 (p)- 
value bound obtained is 269, equivalent to a one-sided 
Gaussian SNR of 19. 

The final analysis was performed after two previous 
analyses. The first analysis involved partitioning the 
trials into randomly chosen sets of four matched trials, 
one at each of the settings choices. The version of the 
distance-based analysis available at the time yielded a 
significantly smaller total violation than the final analy¬ 
sis. The PBR analysis at the time was faulty, but sug¬ 
gested a significantly higher log 2 (p)-value bound than re¬ 
vealed by the final analysis. Because the randomization 
strategy used in this analysis is not acceptable for cer¬ 
tification purposes, a second analysis was performed af¬ 
ter the analysis procedures were updated. During this 
analysis, we discovered that the count-rate variations in 
time significantly affect the log 2 (p)-value bounds, requir¬ 
ing that the analysis be performed adaptively. A choice 
for adaptation parameters was made after investigating 
the timescale of the variations. The estimated total vi¬ 
olation found was the same within error bars as the one 
for the final analysis. The third and final analysis was 
required because we discovered an error in our original 
method for Bell function truncation in the PBR analy¬ 
sis resulting in an overly optimistic log 2 (p)-value. The 
adaptation parameters were chosen for the final analysis 
based on our experience in the second round of analy¬ 
sis. Because of this history, a moderate bias in the esti¬ 
mated total violation and in the log 2 (p)-value bounds is 
expected. 


2. Pulsed Quantum Source 

The data set for this experiment consists of 4450 trials. 
The settings for each trial were chosen randomly. Each 
party’s measurement outcome consists of a sequence of 
time-tagged detections. The source was pulsed with the 
pulses synchronized with a clock whose “ticks” were also 
recorded for each trial. There are approximately 12500 
pulses per trial before preprocessing. The original anal¬ 
ysis of the data reported in Ref. [S] analyzed each pulse 
as a trial. To consider the results of this analysis as ev¬ 
idence against LR requires an assumption such as that 
each pulse is independent and identical. Defining trials so 
that they contain all the measurements that occur while 
the settings are fixed avoids making this assumption. 

The sequences of timetags from each trial were prepro¬ 
cessed as follows: We first corrected for the time-offset 
of A as we did for the data from the continuously emit¬ 
ting source. We then removed detections outside narrow 
windows containing each pulse. The windows were de- 
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termined relative to the recorded clock ticks and have 
a width of 16000 time units. The pulses are separated 
by about 256000 time units. We dropped the first 200 
pulses and saved the 12200 subsequent pulses, dropping 
the rest. To correct for intermittent interference causing 
excess detections, we “blanked out” (removed detections 
in) pulses where there was an excess number (three or 
more) of detections outside pulse windows in the period 
spanning three pulses before and after. Note that the 
preprocessing is local in the sense that the parties can in 
principle perform it without communicating, given that 
they have synchronized clocks. 

As in the case of the continuously emitting source, 
there were several analysis attempts. In the first attempt, 
the order of the trials was randomized and we only con¬ 
firmed that the violation based on distance-based anal¬ 
ysis was consistent with the results reported in Ref. [S]. 
The PER analysis was not performed at this time. Later 
analyses were performed in parallel with the analysis of 
the continuously emitting source, with the final analysis 
correcting the same problem with our original implemen¬ 
tation of Bell-function truncation. 

For the final analysis, we did not perform a version 
of the conventional coincidence analysis as the pulsed 
nature of the source made this unnecessary. Applying 
the distance-based analysis using the distance-based Bell 
functions of Ref. [5] failed to show a violation; we at¬ 
tribute this to the presence of an excess of multiple de¬ 
tections during pulses and the sensitivity of the analy¬ 
sis to detection-rate changes. (We attribute most mul¬ 
tiple detections to local detection artifacts, such as de¬ 
tector after-pulsing, rather than photons. These local ef¬ 
fects confuse the distance-based analysis by adding non¬ 
violating LR signals.) We therefore used a simpler Bell- 
function with no parameters. This Bell-function is ob¬ 
tained by adding the Bell-function derived from the Bell 
inequality used in Ref. [9] over the detections for each 
pulse. For this purpose, multiple detections in a pulse are 
counted as one. This is an instance of a general strategy 
for pulsed sources where the settings are not changed 
for every pulse. Consider a Bell function B for the 
detections from one pulse satisfying the Bell inequality 
{B{T, S'))lj^ < 0, where T is the detection pattern and S 
the measurement settings. If we have a sequence of pulses 
at fixed measurement settings S with detection patterns 
Ti, we can define a Bell function B'{T, S) = B{Ti, S'), 
where T is the sequence of detection patterns Ti. The 
Bell inequality (i3'(T, S))lj^ < 0 is also satisfied by B'. 
To avoid assuming that trials are independent and iden¬ 
tically distributed (i.i.d.), one can analyze the violation 
of B' instead of B. This change of view enables the 
PBR analysis: As noted in Ref. [8], all Bell functions 
for two parties and two settings can be derived from a 
set of distance-like functions satisfying an iterated trian¬ 
gle inequality. This makes it possible to apply the PBR 
analysis as we have done here. Although the parameters 
of the Bell-function require no training, the total viola¬ 
tion was computed using the procedure of App. 0 with 


an initial set of 200 trials set aside for initializing the pre¬ 
dictions. In each step, the predictions in the procedure 
were updated using the previous 200 trials to account for 
experimental drift. The distance-based analysis found a 
total violation of 1.41(18) x 10^, corresponding to a nom¬ 
inal SNR of 7.8. 

For the PBR analysis, we computed the necessary 
truncation and mixture parameters adaptively based on 
the Bell function values obtained in the Bell function 
analysis. The first 400 trials were reserved for training. 
Starting with the 401st trial, we updated the parame¬ 
ters every 200 trials based on the previous 1600 trials (or 
less, initially). We found a log 2 (p)-value bound of 33, 
equivalent to a one-sided Gaussian SNR of 6.3. 


3. Classical Source 

The experiment on the classical source consisted of 9 
groups of four trials at each of the four settings choices. 
The settings were not chosen randomly. Thus the inter¬ 
pretation of an apparent violation requires i.i.d. assump¬ 
tions. Of course, the source was designed to be LR, so no 
real violation can be observed. The data from this exper¬ 
iment was analyzed just once, after the distance-based 
analysis matured. Each trial has approximately 97000 
detections for each party independent of the settings. 
The first group was set aside for training. A first step 
in all our analyses was to determine systematic timing 
offsets and an estimate of the time-jitter, both were done 
by checking timetag differences on apparent coincidences. 
For this source, the timetag differences immediately re¬ 
vealed that there was an “unexpected” pattern in the 
detections. That is, since there was no attempt at hid¬ 
ing that the source was exploiting the coincidence-time 
loophole, the resulting characteristic detection delays are 
obvious. (Ref. [S] demonstrates a simulated source that 
can successfully hide these detection patterns.) 

For the purpose of demonstrating that a standard co¬ 
incidence analysis (windows determined by Alice’s de¬ 
tections) is deceived by this source, we optimized the 
coincidence window as usual on the training set and ap¬ 
plied the coincidence analysis to the rest. The total vi¬ 
olation found was 6.6488(24) x 10® for a large nominal 
SNR of 2781. We optimized the Bell function for the 
distance-based analysis but were unable to detect a vi¬ 
olation. In fact, the estimated total Bell function was 
significantly negative. We cannot exclude the possibility 
that a better choice of parameters for the timetag Bell 
function exists, though we know on theoretical grounds 
that a violation should not be observable according to 
the distance-based analysis. Given the absence of viola¬ 
tion, the PBR analysis is guaranteed to use trivial test 
factors giving a log 2 (p)-value bound of 0. 
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Appendix C: Conservative Estimates of 
Bell-Violation 

For each experiment, the timetag Bell function B used 
has expectations that are related to the violation of a CH- 
type inequality by multiplying the latter by the expected 
number of photon pairs. In the limit of low time-jitter 
compared to the mean photon-pair inter-arrival time, the 
expectations according to B and that expected from the 
CH-type inequality converge. It is therefore worthwhile 
to estimate the expected value of the Bell function for 
comparison purposes. In principle, for each trial, the 
expectation of B is an experimental observable that, if 
greater than zero, witnesses violation of the Bell inequal¬ 
ity associated with B. If the trials are i.i.d., the expec¬ 
tation can be estimated empirically using conventional 
methods. For tests of LR, this is usually done by esti¬ 
mating the settings-conditional expectations of B, which 
are then added. The uncertainty is obtained accordingly. 
When the trials are not necessarily independent or iden¬ 
tical, there may be no single expectation of B to esti¬ 
mate, so the conventional method cannot be used. Here 
we give an alternative that produces meaningful results 
in the general case. It statistically agrees with the con¬ 
ventional method when the trials are i.i.d.: While the 
estimated uncertainties obtained are slightly larger on 
average, they differ by an amount that is comparable to 
the expected statistical fluctuations in the estimate. 

We emphasize that the purpose of these methods is to 
obtain an estimate of a physical quantity and the associ¬ 
ated uncertainty. They do not yield certificates against 
LR (see [S] for a discussion). While we obtain uncer¬ 
tainties that are appropriate for dependent trials whose 
expectations change in time under normal experimental 
conditions, a sufficiently determined adversary can still 
ensure that our uncertainties are overly optimistic. 

Here is the procedure for our method. A mathematical 
discussion follows the procedure. 

1. a. Initialize the running value of the estimated 

total Bell violation by setting &[o] = 0. 

b. Initialize the running value of the estimated 
variance M[o] =0 

2. For each trial result di, 1 < i < N in order, do the 
following 

a. Before considering di: 

1. Predict the settings-conditional expected 
Bell-function values {B{Di)\Si = s) at the 
next trial as 6pred,i(s)- This prediction 
can be based on any information avail¬ 
able before the Hh trial occurred, includ¬ 
ing calibrations, theory and previous trial 
results. Here, Si = {Sf, Sf) are the joint 
settings random variables and Di are the 
random variables whose outcome values 
are the di. 


2. Determine the predicted average 
Bell function violation fopred.z = 
Ved,j(s)Ps, where is the prob¬ 
ability of settings choice s. Note that 
{bpredA^)) = bpred,i IS known exactly 
before the ith trial. 

b. Now consider dp. 

1. Compute 6[i] = + B(di) — 

(^pred,i(Si) ^pred,i)' 

2. Compute M[i] = U[i-i] + {B{di) — 

^pred,z(Si)) • 

3. Report the estimated total Bell violation as &[Ar] 
with an approximate 68 % confidence interval of 
^[Af] i \/ • 

The simplest method for predicting the settings- 
conditional Bell-function expectations in step 2.a.l of the 
procedure is to compute the sample means conditional on 
settings from the first i — 1 trials and the training trials. 
This works well for stable experiments. For the data 
analyses in this paper, we used a segment of recent tri¬ 
als (including the training trials) instead. We formulated 
the procedure for a fixed Bell function, but the procedure 
also works if the Bell functions are chosen adaptively be¬ 
fore each trial. 

Consider a sequence of trials with each trial’s result 
given by di. We now adopt the usual conventions for 
random variables and their outcome values, where ran¬ 
dom variables are capitalized. Thus Di is the random 
variable for the result from the ith trial, and di is its out¬ 
come value in a particular run of the experiment. The re¬ 
sults consist of the measurement outcomes and settings. 
We let and Sf be the respective random variables 
for the measurement outcome and setting of party X in 
the Ah trial. We let D denote the sequence of random 
variables Di. The random variables Di are not neces¬ 
sarily independent, but the distributions of the settings 
are jointly uniform and therefore independent of each 
other. We let iLi_i be a random variable that captures 
the history of events preceding trial including events 
not captured by D but that are relevant to the experi¬ 
ment. In particular, Hi-i determines Dj for j < i and 
may include additional experimentally relevant informa¬ 
tion. The goal is to obtain an empirical estimate of the 
quantity 

N 

kmW = (Cl) 

i=l 

and a confidence interval for this estimate. Here, hi-i 
is the actual value of the history random variable. 
Throughout, we assume that the relevant real-valued ran¬ 
dom variables have finite second moments. We inter¬ 
pret 6[Ar](/i) as the total Bell inequality violation actually 
present in the experiment, which we estimate with 6[j]. 
We do not assume that the outcome value /ii_i is known. 
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just that it is well-defined for a given run of the exper¬ 
iment. Define bi{h) = bi{hi-i) = {B{Di)\Hi-i = hi-i). 
This is the expected value of the Bell function for the up¬ 
coming ith trial, just before the trial is performed. We 
use the following conventions to refer to functions of ran¬ 
dom variables and the random variables defined by these 
functions: Except for the Bell function B, we use lower 
case annotated symbols for the functions. Applying a 
function to a random variable as in the expression bi{H) 
defines a new random variable. To simplify the notation, 
we also refer to this random variable by its upper case 
variant, so that Bi = bi{H). (Here H refers to the full 
history.) The outcome values of this random variable are 
then denoted by bi = bi{h). 

For i.i.d. trials, bi{h) = {B{Di)) and is independent of 
i and h. The sum 6 [ 7 v](^) = can then be inter¬ 

preted as the conventional total Bell inequality violation 
of the experiment, if it is positive. For the empirical 
estimate of b[N]{h) we could compute but 

instead we use the less-noisy estimate 6 [jv] from the pro¬ 
cedure. The estimate is less noisy because we subtracted 
from b{di) the quantity ( 6 pred,i(si) — &pred,i)j whose mean 
is guaranteed to be zero but is expected to be positively 
correlated with the original estimate. 

The first task is to show that is an unbiased es¬ 
timator of b[N]{h) (that is {B^^j) = ( 6 [jv](i/))). We use 
E{A\B) to denote the conditional expectations of A with 
respect to B interpreted as a function of the random 
variable B. The notation (...) is reserved for uncondi¬ 
tional expectations and expectations conditional on spe¬ 
cific outcome values. Since E{. .. | ...) denotes random 
variables, they may occur inside (...). By expanding the 
definition, we have 

- Bpred.O^ • (C2) 


The expectation of Bp^edy — Bp^ed.i is 0 by design, so 


respect to H that should reflect what actually happened 
during the experiment as indicated in the definition of 
&[jv](h). Formally, we seek a bound 5_ for a (conservative) 
confidence interval for b[j^-^{h) — 6 [^](d) that satisfies a 
coverage condition, namely that before the experiment, 
the probability that —A < 6 [Ar](B) — 6 [jv](B) < A is at 
least 68 %. Here A is the random variable with outcome 
values 5. (We could consider —6 as the lower endpoint of 
a one-sided confidence set with no upper bound, in which 
case we require that before the experiment, the proba¬ 
bility that —A < 6 [ 7 v](B) — S[Ar](B) is at least 84%.) 
Because the trials may not be i.i.d., the standard esti¬ 
mates of variance cannot be applied to determine <5. Our 
method yields an estimate of an error bound given rela¬ 
tively mild assumptions and sufficiently large N. 

Let = B{di) - 6pred,i(si) -I- fopred.z be the incre¬ 
ment &[j] — of the estimated Bell violation from the 
ith trial. By design of 6 predi; we have E{Bi\Hi_i) = 
E{B[Di)\H,_^) = h{H). 

We investigate the statistics of the estimation error 
A[jv] = b[N]{D) - Ai, with 

A, = (C7) 

Note that (A^) = 0 and (Aj^v) = 0), so the variance of 
A [AT] is 


Var(A[Ar]) 



(C 8 ) 


Since E{Ai\E[i_i) = 0, the A^ are martingale incre¬ 
ments adapted to the Hi. (For the relevant theory of 
martingales, see Ref. m-) Martingale increments at 
different times are uncorrelated. That is, for i > j, 
(A,A,) = {E{A,Aj\Hj)) = (B(A,|B,)A,) = 0. A con¬ 
sequence is that the variance of the estimation error sat¬ 
isfies Var(A[A/]) = A^y In detail. 


N 


{Bin]) = 

(C3) 

N 

= ^(£;(B(A)|B,_i)) 

i—i 

(C4) 

N 

= HHh)) 

(C5) 

= ^[n]{H)) , 

(C 6 ) 

where the identity {B{Di)) = {E{B{Di)\Hi_i)) fol¬ 
lows from the rules for iterated conditional expectations. 
(This is a special case sometimes referred to as the “law 
of total expectations”.) 

The second task is to determine an approximate 68 % 


confidence interval for 6 [Ar](h) around &[Ar](d). Note that 
the confidence interval is itself a random variable with 


(C 8 ) 

/ ^ 

= ^2(A,A,)-t-/5]A? 

i>j \i=l 

= 

Since we do not know bi{h), we cannot directly com¬ 
pute Si as our estimate of (A||Bi_i = hi-i). But we 
can use the prediction &pred,i of Bi(h) made before the 
zth trial. Recall that the variance of a random variable 
R is the minimum expectation of (i? — m)^, where the 
minimum is achieved by m = (i?). In conditional form, 
this implies E{Al\Hi_i) < E{{Ai — M)‘^\Hi_i) for any 
M that is a function of because A^ is zero-mean 
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conditional on We set M = Bpred.i — Bi and define 

Si — Si TTi — 13{di) ^pred,i(^2)) 

which we can compute from the available information. 
The variance inequality now implies £'(A||iJi_i) > 
so {{[AT] = can serve as a biased- 

high estimate of the desired variance. Formally, 


N 


EA) 

2=1 

(C13) 

AT 

i—1 

(CM) 

N 

i—1 

(C15) 

N 

E(A) 

2=1 

(C16) 

Var(A[Ar]), 

(C17) 


where U[n] is the random variable corresponding to U[n]- 


To justify as an estimated uncertainty requires 

additional assumptions on the random variables. For 
Chebyshev-type inequalities involving variance and a va¬ 
riety of exponential bounds on tail probabilities, bound¬ 
edness of B suffices (and is typically stronger than nec¬ 
essary). But one would like to use appropriate central- 
limit theorems in the same way as for i.i.d. trials. The 
conditions under which such central-limit theorems hold 
are surprisingly broad, but not unrestricted. Ref. [TS] 
has a variety of relevant versions that can be applied in 
non-adversarial situations where the square errors A^ are 
asymptotically well-behaved. We therefore suggest that 
in typical physics experiments with sufficiently many tri¬ 
als without excessive stability problems, the approximate 
68 % confidence interval of the total Bell violation can be 
given as b[N] i ^yu[N]■ We expect this interval to be con¬ 
servative under most conditions even though the trials 
need not be i.i.d. 
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